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ABSTRACT 

A program to teach evaluation skills to 
middle-manageirent acSministrative personnel is described. The course 
consisted of three training sessions totaling 12 hours. The course 
notebook materials presented are: (1) objectives; (2) planning 
educaticmal experiences through systematic procedures; (3) outline of 
materials — what is evaluation?^ where does evaluation fit in 
educational development?^ objectives of evaluation, purposes of 
statewide evaluation, criteria for evaluating proposed project 
evaluation; (4) "From Evaluation Theory into Practice" — excerpt from 
a speech; (5) student Information System — to be used in determining 
student needs, as an aid to program planning; (6) representative 
learning outcomes and possible methods of evaluation; (7) assessment 
planning chart; (8) trait validation procedures; and (9) an annotated 
bibliography. Indicators of Quality, an instrument for measuring 
school quality by observing the behavior that goes on in the 
classroom, is presented in an appendix* The information collection 
instruments attached are a project release, in which participants 
indicated their interests and needs prior to the program, the final 
evaluation of sessions, and a questionnaire for evaluation skills. A 
brochure describing the program is attached. (KM) 
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The State Education Agency (SEA) sought to raise the level of management skills of 
Nebraska educational administrators through the establishment of planning and evaluation 
capability among SEA personnel. The SEA has designated a section for planning and 
evaluation of Federal, State, and locally assisted elementary and secondary programs. The 
section was placed within the Division of Instruction. Planning and evaluation functions 
were specifically requested of the section by both Federal and State education programs. 

After the training sessions in evaluation, it was anticipated that educational program 
personnel will be prepared to exercise evaluation skills and wil! apply these skills in making 
judgments of the program of educational activities appearing in the Federal and State 
proposals which they analyze. Skills in the change-support process as well as the product 
process would aid personnel in determining the extent to which a change occurred as well 
as identifying those elements of the program responsible for fostering the change. 



Through this activity, the people of Nebraska and specifically those responsible for 
education in Nebraska might begin to identify programs through which pupils in the State 
^ school system might make progress toward realization of the Goals for Education. 

^identify Problen 



RESEARCH: The first review of literature dealt with the material on planning skills. This 
review was aimed at identifying the link between planning and evaluation skills. New 
planning materials were also considered for their application to evaluation. This concept was 
supported in a pamphlet developed by the Video In-Service Program at Milford, Nebraska, 
"Planning Learning Experiences Using Systematic Procedures." Their model was also 
dependent upon the inclusion of evaluation procedures within the planning process. 

In examining what evaluation was, it was also linked to decision-making. Specifically, 
educational evaluation is the process of obtaining and providing useful information for 
making educational decisions. After examining the four kinds of decisions which can be 
made (planning, structuring, implementing, and recycling), one could relate the kinds of 
evaluation required for each (context, input, process, and product). Therefore, the 
continuous aspect of the process of evaluation was established. 

More research on the need for evaluation in decision-making pointed to the necessity of 
providing information for planning decisions which must include a component that has as 
Its mission the development of an information system. This requires delineating information 
needs, determining criteria and levels to be u^ed in analyzing information acquired, 
O establishing mechanisms to obtain this information, and building networks to insure the 

JC flow of such information to the appropriate decision-makers. Therefore, an investigation of 
a Student Information System seemed appropriate. 



Evaluation for decision-making at the State level requires more than knowing where 
individual pupils stand. It also requires the evaluation of whole school systems and districts. 
An instrument developed at Columbia University, the Indicators of Quality, was designed to 
measure school quality by observing the critical behavior that goes on in the classroonri. 

OUTLINE: The elements of evaluation found to be most important in the research were 
divided into four general areas. These arftas provided the basic concepts which were to be 
presented in the training sessions. They were: 

A. Using a Systematic Planning Model 

B. Evaluation and Educational Decisions 

C. A Student Information System 

D. The Indicators of Quality 

The outline of this material was of an informative nature. It was a well-organized 
method for presenting the material involved in the key concepts within the notebook. Also 
included in the notebook, was an annotated bibliography of resources used in the entire 
research process. 

OBJECTIVES: On the basis of the key concepts which were identified, specific behavioral 
objectives were written for the participants. The written objectives were to be included in 
the notebooks supplied to the participants at the training sessions. 



Once the problem had been analyzed within the Department, some reinforcement and 
recommendations were needed from outside sources. A Task Force as such, was not formed 
for the whole problem. Instead, consultants in each of the key concepts were called upon 
to discuss that concept with the Planning and Evaluation staff. 

Mr. Carl R. Spencer, Mr. Robert Klabenes, and Mr. Tom Hill, all associated with the 
Video In-Service Program at Milford, Nebraska, were contacted about their manual on 
"Planning Learning Experiences Using Systematic Procedures/' This manual was prepared by 
Mr. Spencer, and Dr. James O'Hanlon, from the University of Nebraska. Their material was 
thought to be appropriate as a "link'' between the planning skills and evaluation skills. 
After meeting with these people and explaining how their material could be used in the 
training sessions, they were very receptive to the idea of presenting it. 

Dr. F. William Sesow of the University of Nebraska was contacted about his work with 
the Indicators of Quality at Columbia University. He was able to describe the instrument 
and supply valuable written material about it. Since he was formerly the curriculum director 
in a Nebraska school system which used the instrument, he was very well acquainted with 
its present and possible uses. He agreed to discuss the Indicators of Quality at the training 
sessions, and suggested that administrators from other school systems using the instrument 
might also be contacted to speak. 



Two Department members were contacted about the other two concepts in the outline. 
Dr. Francis E. Colgan, Director of the Planning and Evaluation Section, agreed to discuss the 
relationship between evaluation and educational decisions. He suggested the content of an 
activity which participants might engage in to put the concept into practice. After identifying 
the goals for education toward which each participant's work was aimed, they would be 
responsible for deciding what functions they perform to meet those goals, what kind of 
decisions the functions led to, and the evidences of their success. 

Mr. Joe Mara, Consultant in Pupil Assessment, agreed to discuss the Student Information 
System (SIS) at the training sessions. Both he and Dr. Colgan were acquainted with the SIS 
and had discussed it with one of its developers, Mr. Bruce Wainwright of Utah. Mr. Mara was 
especially interested in explaining possible uses of the SIS by Department members. He con- 
tributed written material on the validation procedures of a scale that could identify a given 
pupil trait, and would explain this process at the sessions. It was suggested that the activity 
related to educational decisions be extended to include the SIS questionnaire items. 

It was decided that the written outline of the key concepts was to be placed in a notebook 
for each participant. This notebook would also hold the annotated bibliography, activity 
charts, schedule of events, and supplementary material. In preparation for the activity, a 
project release would need to be designed to group participants. Because of the somewhat 
negat'^'e response to the project releases for the planning skills, only one would be used for 
evaluation skills. The notebook would also be compiled and given to participants prior to the 
sessions. This decision was also based on responses to the evaluations of the training sessions 
on planning skills. 



The project release was distributed to participants prior to the training sessions. Given a 
list of educational concerns of the people of Nebraska which could be considered as goal 
statements, participants were asked to check the level of concern in their work with what 
the statements suggested. In this way participants could be grouped according ^o the goals 
toward which they work. They were also given their notebooks on evaluation skills so that 
they could be examined prior to participation. 

Final plans for the presentations were set with the individuals consulted about each of 
the key concepts. The presentation of the Systematic Planning Model was set for the first 
afternoon of the sessions. The presentation included a slide-tape program explaining the 
need for systematic procedures in planning learning experiences. This need is based on the 
assumption that society has a right to hold the leadership of a public school accountable for 
the quality of instruction in that school. Once the need for goals in a program is 
established, small groups would be formed to work on a programmed series of steps to meet 
their goals. Thu^, they would actually be involved in using the systematic procedure. 

The morning of the second session was to present the need for evaluation in making 
educational decisions. This would be explained by Dr. Francis Colgan before small group 
activities were begun. The small groups would be formed on the basis of the project releases 
which were returned. Each group would focus on one goal for education. An Assessment 
Planning Chart would be devised so that participants could identify their functions and 
^ classify them under the kinds of educational decisions which they exhibit. An investigation 

||^|^(^" of the evidence which they have of their success would then be necessary. 



After an explanation of the Student Information System by Mr. Joe Mara in the 
afternoon session, participants would be asked to continue working with the Assessment 
Planning Chart. First, they would identify those evidences of success which result in 
pupil-behavior changes. Then they would get a chance to use two types of questionnaires 
used in the SIS. They would be asked to identify those items on the SIS questionnaires 
which they felt might indicate the desired behavior changes. 

The morning of the last session was to be devoted to an explanation of the Indicators 
of Quality by Dr. F. William Sesow. Dr. Clifton Robinson was asked to discuss his 
experience with this instrument in his school system in Nebraska. Their presentations would 
deal mainly with what the instrument is, how it is administered, the kinds of results 
obtained, and what these results mean to a school system. !t is based on four characteristics 
of school behavior that are judged to be basic to quality: individualization, interpersonal 
regard, creativity and group activity. There may be other "indicators," but these four 
appear to be the basic ores. It was ascertained that results of the application of this 
instrument could serve in an important way in the diagnosis of areas for the next steps in 
school improvement. 

A time schedule including the above speakers, activities, and presentations was formed. 
The training sessions were scheduled for the auditorium at the Department of Roads on 
November 23, 1970 (1:00-4:00), November 24, 1970 (9:00-4:00), and November 25, 1970 
(9:00-12:00). 



EVALUATION OF THE LEARNING PACKAGE ON SYSTEMATIC PLANNING: In an 

attempt to develop an exportable package concerning systematic planning, part of this 
presentation was mediated through Department facilities. Since this was the first showing of 
the mediated package, critical analysis of the presentation was desired by the Department, 
as well as those who prepared the content. Using a Likert Scale of 1-7 (low - high), 
participants were asked to rate the adequacy of the techniques of presentation, (various 
written materials, audio-visual materials, and small group activities) and content. The scale 
failed to yield any discrimination among responses. Responses generally clustered at the 
Q "high end" (5-7) of the scale. 
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PROGRAM QUESTIONNAIRE: Several weeks after the training sessions, questionnaires 
over the content of the sessions were distributed to participants. Ail questions dealt with 
material from the notebooks on key concepts, or the major ideas expressed by the speakers. 
Part I included a list of functionr which occur within the Department. Participants were 
asked to code the kind of decision and evaluation which that ^unction exhibits in 
educational development. Part II gave a list of classroom situations and asked participants to 
identify the concept which the concept suggested. The concepts were representative of the 
four Indicators of Quality. Part III included a list of true or false statements about the key 
concepts. The purpose of this questionnaire was to determine the material which wasn't 
adequately learned or retained through an item analysis, and then to determine the 
correlation between the results and the evaluation forms. 

FINAL EVALUATION: This evaluation was distributed several weeks after the training 
sessions. Part I contained two lists. The first list was of the key concepts covered in the 
sessions. Participants were to indicate the most and least useful topics, as well as those they 
would like more or less information about. The second list was composed of the methods 
used — media, notebooks, group aiscussi' s, department speakers, and outside speakers. 
They were then asked to indicate me most and least effect - used, as well as what should 
be used more often or eliminated. Part II contained several open-ended questions which 
dealt with problems in using the material presented, participants' contributions, and most 
meaningful material. Their answers could thus be categorized and ranked by percentages. 



Monday - November 23, 1970 (1:00-4:00): The first session began with twenty-four 
participants present. Several people directly involved with evaluation procedures within the 
Department were added to the list of participants in the planning skills. They were 
instructed to bring their notebooks with them. The afternoon's activity was based on the 
key concept. Using ? Systematic Planning Model. Outside resource people present for the 
presentation were Mr. Bob Klabenes, Mr. Tom Hill, and Mr Carl Spencer. 

A slide/tape presentation introduced this concept. It explained the need for using 
systematic procedures in planning learning experiences. This iieed was based on the 
assumption that society has a riglit to hold the leadership of a public school accountable for 
the quality of instruction within that school. Leaders must not only be responsible, but 
accountable, also. After this premise had been established, participants chose goals on which 
they would like to work in small groups. Using a programmed manual, they were to plan 
learning experiences to achieve their goals. 

This nrx)del was ideal for relating evaluation skills to planning procedures. One of the 
specific steps in the planning model provides for planning evaluation procedures. The major 
O ^ advantage of this mediated package and manual was to actively involve participants in the 

s^C systematic planning procedures. An evaluation of the package was made by participants 

upon completion of the afternoon session. 



Tuesday - November 24, 1970 (9:00-4:00): Dr. Francis E. Colgan, Director of the 
Planning and Evaluation Section, began the morning by explaining the need for evaluation 
in making educational :.!ecisions. The four kinds of educational decisions for educational 
development require four special kinds of evaluation. Small groups were then formed on the 
basis of the project release. Each group worked on the tentative goal for education which 
they thought they functioned under most. Each participant was furnished with an 
assessment planning chart in his notebook. The first column asked for the goal statement, 
while the second listed the four kinds of educational decisions under which they were to 
identify their functions. The final column asked for evidence of success in each function. 

After lunch Mr. Joe Mara, consultant in pupil assessment, explained the Student 
Information System developed in Utah. Quest'onnaires which accompany the SIS were 
examined. The items on the questionnaires can be used to measure behavioral traits in 
students. Participants were then asked to choose those item': which could measure the 
evidences of success which were exhibited in behavior changes of the pupils. Such an 
information system is important to an evaluation department in making educational 
decisions. 

Wednesday - November 25, 1970 (9:00-12:00): The morning of the last session was 
concerned with the evaluation of school systems. Dr. F. William Sesow from the University 
of Nebraska introduced the Indicators of Quality. This instrument assumes that there are 
four basic indicators. They are individualization, interpersonal regard, creativity, and group 
activity. There might be others, but these are seen as basic. The developers of the 
instrument at Columbia University wanted to know what those indicators looked like — 
what teachers and students did when they were exhibited. The results contained 51 
polarized items which trained observers look for in the classroom. Either the item is 
exhibited or it isn't. This evaluation system is of a school district and not a class or a 
teacher. Dr. Sesow was an administrator in the Bellevue Public Schools while the instrument 
was being used. Dr. Clifton Robinson of the Omaha Westside Community Schools also 
related their experiences with the instrument. Both speakers saw this as an excellent 
opportunity to evaluate the environment of the classrooms within a school system. 



PROGRAM QUESTIONNAIRE: After doing an item analysis on the questions given 
several weeks after the training sassions, conclusions could be drawn about the content of 
the topics presented. Two kinds of results were significant — those questions missed by no 
one and those missed by more than 30%. There were seven out of twenty questions which 
were missed by no one. They were concentrated in the parts on identifying Indicators of 
Quality and the true or false statements about all the concepts. There, were several questions 
missed by 30% or more. All but one dealt with identifying the kind of educational decisions 
and evaluation exhibited in various Department functions. 

PROGRAM EVALUATION: 
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Most participants thought that time would be their biggest problem in using what they 
learned. Very few participants felt they had contributed anything except in group 
discussions. The speakers and the presentation of the Indicators of Quality rated highest in 
the category of most meaningful. 



This publication is the result of an effon. by the Nebraska State 
FOREWORD Department of Education to raise the level of management skills of 

Department personnel. The State Education Agency designated a section 
for planning and evaluation within the Division of Instruction. Planning and evaluation 
functions were specifically requested of the section by both Federal and State education 
programs. 

The Planning and Evaluation Proposal states the goal of this project as follows: "To prepare 
training materials and training opportunities for educational middle-management personnel 
in the process of evaluation." 

Twelve hours were specifically provided and used for the training sessions in evaluation 
skills for program administrators of the Instructional Division. This publication includes the 
materials supplied to the participants in a notebook for the sessions. It also includes the 
instruments that were used to collect information from the participants. The process and 
evaluation of these training sessions appear in a companion piece to this publication.lt is 
anticipated that divisions in other Departments of Education, as well as local education 
agencies, might be able to profit from the process and materials used in Nebraska for 
training middle-management personnel. 

The e$;ablishment of evaluation skills as an operational function among program 
administrators in the Division has been a sincere effort to raise the level of accountability 
toward that which is require'^ in the investment of Federal, State, and local dollars. We 
believe that these sessions have contributed to that effort. 



Cecil E. Stanley 

Commissioner of Education 
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Suggested Program Schedule for 
Training Sessions in Evaluation Skills 



Monday 

1:00 - 4:00 Using a Systematic Planning Model 
Tuesday 

9:00 ' 9:30 Evaluation and Educational Decisions 

9:30 - 11:00 Part I of the Assessment Planning Chart 

Lunch 

12:00 - 1:00 Rationale and Description of a Student Information System 

1:00 - 2:00 Part II of the Assessment Planning Chart 

2:00- 2:15 Coffee Break 

2:15 - 3:15 Discussion of the Assessment Planning Chart 

3:15 - 4:00 Trait Validation Procedures 

Wednesday 

9:00 - 12:00 Indicators of Quality 



Hi 



NOTEBOOK MATERIALS 
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OBJECTIVES: 

1. The participants should be able to state the goals for education by which their work 
should be influenced. 

2. They should be able to identify the functions they pursue which relate to those goals. 

3. The participants should be able to describe the four types of decisions necessary to 
educational development, and the kinds of evaluation which each require. 

4. They should be able to identify the kinds of decisions which are necessary for their 
functions. 

5. They should be able to name the kinds of evidence whicn are needed to judge their 
success. 

6. They should be able to relate the outcomes of their work to the goals for education 
which are influencing their work. 

7. The participants should be able to demonstrate several uses of the Student Information 
System. 

8. They should be able to apply the Systematic Planning Model to evaluation procedures. 

9. They should be able to identify and interpret the four Indicators of Quality. 
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PLANNING EDUCATIONAL EXPERIENCES THROUGH SYSTEMATIC PROCEDURES 



Carl Spencer 
Consultant, Milfora Service Unit 

A. Rationale 

1. Professional education managers must be concerned with the ultimate purpose, 
effectiveness, and efficiency of the program. 

2. Specific answers to the following questions are needed to provide an effective and 
efficient program: 

a. What is the program to produce or achieve? 

b. How do you know the objectives are of importance for the program? 

c. How will one know when the program has achieved its ultimate purpose or 
goal? 

d. How will one measure the effectiveness and efficiency of the program in 
reaching its goal? 

e. How will one know when to revise or change the program design and goals? 

3. The professional educator's primary attention should be given to: 

a. The student/learner. 

b. His predictable learning success. 

c. His learning of skills, knowledges, and attitudes which are relevant to the role 
for which he is being prepared as a citizen, parent, producer, and consumer. 

4. Requirements for designing and implementing instructional programs and 
coordinating methods for managing these programs: 

a. Start with the learner. 

b. Design programs for instruction and management measured in terms oi 
student learning or success. 

c. Establish measures of accountability for teachers, administrators, managers and 
learners which must be expressed in terms of specific performance, thereby 
allowing an objective evaluation of performance efficiency for all concerned. 

d. Provide for plans of action which assure the continuous production or 
development of relevant learner behaviors (skills, knowledges, and attitudes) 
assuring later success for the learner. 

e. Provide plans of action which efficiently and effectively integrate all members 
of the instructional and management teams while providing the greatest 
societal benefits at the least cost. 

f. End with the learner as the educational referent for planning, designing, 
implementing, evaluating, and managing of instructional and management 
programs. 



5. Educators must establish more systematic procedures in order to satisfy this 
criterion of accountability to both learners and society. 

a. Producing predictable learner achievement requires a more objective and 
controlled process for: 

(1) Defining relevant skills, knowledges, and attitudes to be produced. 

(2) Defining the "learnings" to be produced in measurable terms in order to 
assess learner achievement. 

(3) Specifying in advance the performance requirements representing the 
ultimate learner achievements to be measured on completion of instruc- 
tion. 

b. Formal methods for planning, designing, implementing, evaluating and 
managing plans of action must be substituted for informal and less efficient 
methods. 

c. The formal method best suited to existing requirements in education and 
training is that of a system approach to educational problems. 

The Systematic Planning Model (James O'Hanlon and Carl Spencer) 

1. Any learning experience must be based on some concern felt by the learner. 

2. Establishing specific objectives 

a. What will the learner be able to do to demonstrate that he has reached the 
goal? 

b. The objectives should be closely related to one of the learner outcomes, stated 
as an observable behavior, and measurable. 

3. Assessing the learner's status 

4. Planning learning activities 

a. Set 

b. Modeling 

c. Appropriate practice 

d. Reinforcement 

5. Planning follow-up experiences 

3. Helping learners apply what they've learned. 

6. Planning evaluation procedures 

a. Did the learning succeed in getting the learner to incorporate the objective 
behavior in his work? 

b. What parts of the learning experience were most helpful to the learner? 
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To operationalize evaluation, we need objectives. 
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OUTLINE OF MATERIALS 



i. What is Evaluation? 

A. Evaluation is a process providing some basic elements to administrators: 

1. An internal info'-mation system for better accomplishing the process of 
adaption in the course of change. 

2. Feedback - information which is intended to effect control. 

3. Information to decision-makers. 

B. Educational Evaluation is the process of obtaining and providing useful 
inforifijtion for making educational decisions. 

1. Process - A specific, continuing activity which includes many methods and 
involves a number of steps or operations. 

2. Obtaining — Making available through processes (collecting, organizing, 
analyzing and reporting) and forma! means (statistics, measurement, and 
computer processing). 

3. Providing — Fitting together into systerr3 or subsystems that best serve the 
needs or purposes of evaluation. 

4. Useful — Appropriate to predetermined criteria evolved through the 
interaction of the evaluator and the client. 

5. Information — Descriptive or interpretive data about things that are real and 
their relationships. 

6. Educational Decisions — Choosing among alternatives which affect education. 

C. Evaluation is the process of providing and using information to make planning, 
structuring, implementing, and recycling decisions. 

11. Where Does Evaluation Fit in Educational Development? 

A. Evaluation may be one element within educational development. 

1. Educational needs 

a. Of the society 

b. Of the individual 
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2. 



Specify the problem 

a. Problem 1 — need 

b. This is not a problem that exists as an obstacle. 



3. Conceptualization of a strategy 

4. Implementing various types of strategies 

5. Developing an operational plan 

6. Developing processes, procedures, and materials 

7. Development of an evaluation design 

8. Testing the treatment 

9. Analysis, interpretation, and recommendation 

Evaluation may be a part of all the elements of educational development. 

1. Planning decisions require context evaluation. 

a. These decisions specify changes needed in a program. 

b. This type of evaluation is aimed at continuous monitoring of the program 
to identify needs as they occur. 

c. The method of this evaluation is the definition and frequent measurement 
of variables which characterize ^he total program of interest. 

2. Structuring decisions require input evaluation. 

a. These decisions specify objectives; method, personnel, budget, schedule, 
organization, and content for use in effecting desired changes. 

b. Input evaluation is aimed at identifying and assessing relevant capabilities 
available to the program to define objectives, and to identify and assess 
alternative action plans for achieving the stated objectives. 

c. The method used in this evaluation is not specified. The prevalent 
practices, however, include panel deliberations, appeals to professional 
literature, and employment of consultants. 

3. Implementing decisions require process evaluation. 

a. These are the decisions used in carrying through the action plan. 

b. Process evaluation is aimed at detecting or predicting defects in the 
project operations and/or project design. 

c. The method used in this evaluation includes continuous identification and 
monitoring of the potential sources of failure in a project. 

4. Recycling decisions require product evaluation. 

a. These are the decisions used in determining the relation of outcomes to 
objectives and in determining whether to continue, terminate, evolve, or 
modify the activity. 

10 



b. Product evaluation is aimed at relating outcomes to objectives and to 
context, input, and process. It is aimed at the measurement and 
interpretation of outcomes. 

c. There are three basic methods used in this type of evaluation: 

(1) defining and measuring criteria associated with the objectives of the 
activity. 

(2) comparing these measurements with predetermined, absolute or 
relative standards. 

(3) making interpretations of the outcomes, using the recorded context, 
input, the process information. 

5. Evaluation is a continuous process in this plan which may be initiated at the 
following times: 

a. prior to decision to write a proposal 

b. while writing a proposal 

c. during project implementation 

d. following or near the end of a complete cycle of the project 

Hi. Objectives of Evaluation 

A. Providing data related to system operation for administrative direction and control 
of change. 

B. Providing for the collection, organization and storage of data relative to programs 
and goals of the system. 

C. Providing for the analysis and evaluative procedures and techniques for program 
implementation, operation and termination. 

D. Providing a response capability for the facilitation of change and revision of 
programs and procedures. 

E. Providing for an ongoing evaluation of the evaluation system. 

F. Providing for the communication of information within and without the system. 

IV. Purposes of Statewide Evaluation 

A. To provide basic information for helping the student assess his own progress 
through the educational systems of the state, so he can understand himself, his 
educational needs, and his future possibilities. 

B. To provide the teachers and administrators in every school system with basic 
information for assessing the effectiveness of all principal phases of their 
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educational programs in sufficient detail to indicate specific steps to strengthen 
those programs. 

C. To provide the state education authority with basic information needed for 
allocating state funds and professional services to equalize educational 
opportunities for all children in the state. 

D. To pro»/ide local and state research agencies with data for generating and testing 
hypotheses concerning the improvement of all aspects of the educational process. 

E. To provide every school system with strong incentives to experiment, under 
controlled conditions, with new and promising educational programs, materials, 
devices, and organizational arrangements. 

F. To periodically provide the state legislature and the general public with readily 
interpretable information concerning the progress of the state system of education 
as a whole and of each local system. 

Criteria for Evaluating Proposed Project Evaluations 

(Note-^ The first four are general and common to any good evaluation information.) 

A. Internal Validity - information provided by the evaluation must show a reasonable 
correspondence to the phenomena which it purports to describe or interpret. It 
must have fidelity — be true. 

B. External Validity - the information must be generally transferable to similar 
situations beyond the one in which it was collected. 

C. Reliability - a repetition of the evaluation should produce essentially similar 
findings. 

D. Objectivity - the data should be interpreted in much the same way by a number 
of people. 

Note: In addition to these first four, evaluative information requires: 

E. Relevance - the information must relate to the decisions to be made. 

F. Significance - the information must be weighted for its meaning in relation to the 
decision. Not all relevant information carries the same weight. 

G. Scope - the information must relate to all aspects involved in the decision. 

H. Credibility - the evaluator who secures information must be trusted by the 
decision-maker and those he must serve. 

12 



I. Timeliness - the information must be prep?>'-**d in time to be useful to the 
decision-maker. 

J. Pervasiveness — the information must get to all who need it. 
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"FROM EVALUATION THEORY INTO PRACTICE" (excerpt) 

Howard 0. Merriman 
Director of Evaluation and Research 
Columbus, Ohio, City Schools 

The evaluator's role in context evaluation may be largely one of delineating the 
information needs of decision-makers, establishing the criteria by which exceptions to the 
acceptable level of performance or behavior can be identified, developing the mechanisms 
for collecting this information, and providing the information to the appropriate 
decision-makers, decision influencers and audiences in such a way that it can be understood 
and utilized. Much information is available within school districts which could be of high 
utility in making decisions. However, access to such information in a timely manner may be 
restricted through a lack of knowledge of its availability or applicability, lack of 
data-handling capability or capacity, or through a lack of established networks for 
information flow. 

The evaluator works at two levels: (1) establishing a context evaluation information 
system and (2) facilitating the flow and use of such information. The former, establishing 
the system, is a means for the evaluator to employ the leverage principle, thereby 
multiplying his effectiveness and gaining efficiency. The evaluator's interest is in delineating 
information needs, and the establishing of a system to obtain and provide that information. 
He must utilize other roles in the organization to operate that system; for example, the 
teachers and counselors who administer tests which provide data on individual students and 
groups of students. The processing of test answer sheets by a data-processing unit is a part 
of obtaining information, just as the printouts of school summary dats are a part of 
providing information. Neither of these operations needs to be maintained or operated by 
the evaluator. He is, however, interested i». setting up the process or system whereby the 
data is obtained and .provided. He will wish to test the information system for quality 
control, since the reliability, validity and precision of the information provided through that 
system are very important to him. 

The evaluator, while facilitating the flow of useful information must, at the same time, 
avoid overburdening the decision-maker with more information than he can use. He may 
establish methods of exception reporting, for example, through which the decision-maker 
receives information only v/hen previously established upper and/or lower limits have been 
exceeded. 

Once such a context evaluation in.ormation system is established, a major role of the 
evaluator may be that of resetting criterion levels or establishing new criterion measures 
which will reflect new conditiors of concern to the educational community. 

In summary, an evaluation department which is to provide information for planning 
decisions must include a component which has as its mission the development of an 
information system by: delineating, v/\th decision-makers, their information needs; 
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determining criteria and levels to be used in analyzing information acquired and establishing 
mechanisms to obtain this information; and building networks to insure the flow of such 
information to the appropriate decision-maker(s). Ideally, over the long run, once such a 
system is established, the evaluator may only have to exercise a quality control function. 



16 



STUDENT INFORMATION SYSTEM 



The gathering of student information, keeping it updated, and using it in a meaningful 
way to assist students in learning has challenged educators for many years. The Student 
Information System (SIS) is an effort to meet the challenge in the computer age. In the 
past we have been forced to look only at those learning objectives for which we had 
measures - IQ and achievement tests. But now we are starting to realize, in fact being 
forced to realize, that we have a number of objectives, not just scores on tests. In 
behavioral research, learner differences must be identified, as well as attitude highs, IQ 
levels, and all the other aspects of the student. The objectives that are used, must all be 
identified, and program ties used to attain those objectives must be determined. Then it's 
not a matter of one school against another, or one program against another program, but 
rather which kind of things work for which students and under which conditions. Then it's 
not a matter of whether a teacher is "good" or "bad," but rather, what specific skills he 
has that are effective and for which kinds of students. 

The first step was to search the research literature and select the variables and 
information readily available in the schools which correlate in any way with learner and 
program objectives at various levels. Personnel with skills in system analysis, data processing, 
school administration, teaching, learning theory, psychology, social work, occupational 
guidance, personality theory, educational measurement, curriculum, and so on, all par- 
ticipai . \ in the development of SIS. The second step was to determine the best sources of 
the needed information. They are the teacher, parents, students, the various specialists, and 
current school records. After the content was outlined and the sources described, 
instruments (check lists, questionnaires, etc.) were developed for field trials for gathering 
and analyzing data. 

The final step was to determine the best ways to analyze the data so that it would be 
meaningful and useful, and to organize distribution techniques to teachers, administrators, 
etc. A method was devised whereby the instruments can be scored either by hand or with 
an electronic optical scanner for computer analysis. The computer output of SIS includes 
administrative reports as well as records, evaluations, and diagnostics of individual students 
or any particular group of students. 

The primary purpose of SIS is to provide schools with the necessary materials and 
step-by-step procedures that will: 

1. Help determine each student's individual learning and program needs. 

2. Provide teachers, administrators, parents, and students with descriptions of these 
needs. 

3. Provide reports which aid in specifying program planning to meet student needs. 
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4. Provide baseline information to evaluate attainment of learning and program 
objectives. 

5. Provide evaluation data on the effects of any project or program on different kinds 
of student performances for any student group. 

6. Provide necessary data for administrative reports and for program planning. 

7. Provide easily accessible and easily interpretable, permanent records for specific 
target groups of students, comparison of test scores, expected percentages of 
students needing special education, demographic analysis, etc., for long-range 
planning. 



8. Identify incipient learning problems and 
and kindergarten levels. 

9. Provide the basis for an analysis 
parent/school/student expectations of 
administrative problem in many districts.) 



developmental difficulties at preschool 

of agreement/disagreement between 
the educational system (a major 



Information for SIS is provided by a set of data forms which gather information from 
students, parents, teachers, and specialists. These instruments contain information similar to 
the current Cumulative Record, and therefore, could eventually replace it. The basic forms 
are: 

1. Parent Questionnaire — The parent describes the environment, interests, abilities, 
and characteristics of his child. 



2. Student Check Lists — There are three check lists through which the teacher 
describes the achievement, problems, and behavioral characteristics of the student. 

3. Student Questionnaire — The student describes his interests, attitudes, plans, and 
so on. Three levels are available - elementary, secondary, and post-high school. 

4. Special Forms - For gathering test scores, administrative information, specialists' 
report, and the like. 

5. Other Forms - May be developed for individual district requirements and added to 
the system to meet specific district needs. 
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REPRESENTATIVE LEARNING OUTCOMES AND POSSIBLE 
METHODS OF EVALUATION 



LEARNING OUTCOMES 



METHODS OF EVALUATION 



A. Application 


A. Objective Test 


Concept Acquisition 


Product Evaluation 


Memorization of Facts 


Rating Scale 


Problem Solving 


Checklist 


Reading Comprehension 




Skills (number, etc.) 





B, Performance 



Rating Scale 
Checklist 

Product Evaluation 



Classroom Behavior 



C. Rating Scale 
Checklist 

Attendance Record, etc. 



Interest 



D. Questionnaire 
Checklist 

Interest Inventory 
Factual Vocabulary Test 



E. Attitude 



E. Rating Scale 
Questionnaire 
Checklist 
Objective Test 



F. Aspiration Level 



G. Adjustment 



Rating Scale 
Interview 

Simple Objective Test 
Word Association Test 
Open Ended Sentences 



Rating Scale 
Anecdotal Report 
Interview, Sociogram 



The methods of evaluation would be simplified and the process facilitated if one instrument 
were designed to coordinate all the above methods. 
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Instructions for Assessment Planning Chart 



Given the general concerns in education that the people of Nebraska seek for their 
children, department members will decide which goals are appropriate to their positions. 

Working in groups related by common concerns, participants will write functions of 
their work that relate to that goal. The functions will be placed under the kind of decision 
(planning, structuring, implementing, or recycling) which the function exhibits. Participants 
will then be asked to write down what kind of evidence is needed to show if they have 
been successful in that function. For each function, which involves changes in pupil 
behaviors, participants should list the desired behavior change. Each behavior should be 
listed separately and should describe specifically what the pupil should do at the conclusion 
of the project. 

Traditionally, one or more of the methods or instruments of evaluation listed on the 
previous page could be chosen to evaluate the presence of the desired behavior change. But 
assuming that we have one instrument which could be used to evaluate such changes (the 
Student Information System), then participants will be asked to choose items on the 
Confidential Student Questionnaire (Level II) Part II which they feel would measure this 
behavior change. The numbers of the chosen items should be recorded under the final 
column in the square joined to the behavior which it tends to measure, along with the 
numbers of items from the Student Behavior Check List which participants feel also 
measure that trait. 
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TRAIT VALIDATION PROCEDURES 



Before any scale which attempts to identify creative students can be developed, the 
goals and objectives of the programs developed for such students must be determined. These 
goals and objectives must be continLally referred to in the consideration of both students 
and scale items. It is also important that people involvea in the selection of scale items 
understand the significance of each item. It is difficult to indicate that most items relate to 
a specific attitude or behavior. Rather, many items are included because different groups of 
students respond to them with different kinds of answers. If one item reads "I hate dogs," 
the important consideration is not whether an individual likes or dislikes dogs. Rather, the 
usefulness of the item lies in the tendency of certain types of students to give either a 
"high" or "low" response to it. This is known as the "discriminating power" of the item. 

There are two basic steps in developing a scale for the identification of those students 
who might benefit from a program designed to help develop creative potential. In the first 
step, half of those people who are responsible for identifying such students actually fill out 
the Student Questionnaire Level II (SQ fl) themselves. They respond to each item as they 
feel a student who possesses creative potential would. The items which are commonly 
responded to as "high" or "low" by this group, then become those items which a creativity 
scale might contain. 

The other half of this group should respond to each item on the SQ II as they feel a 
student of very limited creative potential would. The items which they agree such a student 
would make a "high" or "low" response on may also become components of a creativity 
scale. Both halves of the group should then review each item on the SQ II to determine the 
degree to which it might relate to the goals and objectives of the program for which 
potential students are being selected. 

In the second step, teachers or other individuals in a position to make student 
evaluations identify a group of students who most nearly exhibit the behaviors associated 
with mastery of the objectives of a creativity program. They are also asked to identify a 
group of students who seem to have very little creative potential. The answers these two 
groups of students made on the SQ II are then analyzed to find on which items their 
responses were consistently different. Each item on which these groups consistently differed 
in their responses is a potential part of the creativity scale. In this process it is very 
important that those people who identify individual students limit their consideration to 
indicators of creative potential alone. Other factors of the student's personality or 
background (personal habits, appearance, etc.) must be given absolutely no weight. 
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ANNOTATED BIBLIOGRAPHY 



Beatty, Walcott H., Chairman and editor, Association for Supervision and Curriculum 
Development, NEA, Improving Educational Assessment and an inventory of Measures 
of Affective Behavior, ASCD, 1969. 

The first part of this book is a collection of four papers which analyze the system we 
have for collecting and using data and which propose extensions of the system to catch new 
purposes and new dimensions. An important chapter by Daniel L. Stufflebeam discusses 
evaluation's role in decision making. The last part is an annotated resource list to enable us 
to measure affective behavior. 

Bush, Donald 0., and Willard G. Jones, "Educational Development — from research to 
practice. . .," Rocky Mountain Educational Laboratory, Inc., Greeley, Colorado, April, 
1969. 

This document discusses the nature and process of educational development. One of 
the most benef>;:ial aspects of the booklet is a presentation portraying a model for this 
development with very precise definitions. It locates evaluation as a necessary element of 
the process being both dependent on previous elements and depended upon. 

Davis, Joseph L., and Martin W. Essex, Educational Evaluation, Columbus, Ohio, 1969. 

This book is a collection of speeches given at a conference sponsored by the Ohio 
Department of Education. They deal with the need and importance of accurate measuring, 
appraising and assessing instruments, as well as evaluation designs an^ models and the use of 
evaluation evidence. 

Dyer, Henry S., "Statewide Evaluation - What Are the Priorities," Phi Delta Kappan, June, 
1970. 

This article does an excellent job of stating the purposes of statewide evaluation. These 
purposes are then ranked by the author in order as he sees their importance with individual 
justification of those ratings. 

Foley, Walter J., "The Future of Administration and Educational Evaluation," Educational 
Technology, July, 1970. 

This article discusses the change concept of educational systems. It differentiates 
between the administrative function (managing change) and the role of evaluation (providing 
information). With this difference in mind he states the objectives of evaluation and designs 
a model evaluation system. 
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Hemphill, John K., ' Educational Development," draft, February, 1969. 

This article d stinguishes between two types of educational development; the product 
development approach and the change support approach. The emphasis of change support 
approacf is on the continuation of educational development as a process. It is never to be 
completed since it can never be said to reach a point where further improvement is not 
possible. Objectives then become temporary states and evaluation becomes feedback. 

Merriman, Howard O., "From Evaluation "»"heory Into Practice," Journal of Research and 
Development in Educatior , Vol. 3, No 4, Athens, Georgia, Summer, 1970. 

This article emphasizes the four kinds of decisions made in educational development 
and the kinds of evaluation they require. Each type of evaluation is then described in 
relationship to the information it will produce in order to arrive at one of the four kinds of 
decisions. 



Olson, Thomas A., and Leiia Marvin, "Evaluation: One State's Approach," American 
Education, Vol. 6, No. 4, May, 1970. 

This article describes the process used by the Department of Education in Illinois tc 
evaluate their 55 Title III projects. Their major attempt was described as "humanizing 
evaluation through consultants." Contrary to most evaluation techniques, they were 
enccji'aged to make value judgments and offer personal opinions. 



Provus, Malcolm, "Toward a State System of Evaluation," Journal of Research and 
Development in Education, Vol. 3, No. 4, Athens, Georgia, Summer, 1970. 

This article discusses the cycle of ati evaluation system and the areas deserving special 
emphasis. The whole system is based on the idea tnai one plans on the basis of knowledge 
derived from evaluating one's successes and failures. The a^jthor also gives a list of questions 
which must be answered to make the system work. 



Spencer, Carl R., Planning Workshops Through Systematic Procedures, pamphlet, ASCD 
25th Annual Conference, March 16, 1970. 

This article provides a rationale for using systematic procedures in planning educational 
experiences. The pamphlet then follows up the rationale with the actual model for 
systematic planning. This model provides for evaluation procedures when it establishes 
objectives. 



Stuff lebeam, Daniel, Robert Hammond, and Egon G. Guba, "Planning and Implementing 
Title III Evaluations," United States Office of Education, April, 1968. 

This statement begins with a discussion of the variables of the three dimensions of 
evaluation-instruction, institution, and behavior. They define evaluation in terms of 
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providing information for four types of educational decisions. They describe the kinds of 
evaluation necessary for each of these decisions in terms of objectives and methods. 

Wainwright, Bruce, Manual of Student Information System, Lincoln, Nebraska, April, 1970. 

This manual explains the Student Information System (SIS), its purpose, how it was 
developed, its structure and content, and all possible uses. It includes information on 
administering the questionnaires, as well as scoring methods and interpretation. 

iMew England Educational Assessment Project, A Guide to Assessment and Evaluation 
Procedures, Providence, Rhode Island, February, 1967. 

This guide is aimed at aiding local school systems in *he complex work of evaluation of 
Title I and other special education projects. It provides che general design of an evaluation 
program and strongly suggests that the instruments used for measurement are pertinent to 
the project activity and to the pupils involved. 

Office of the Utah State Board of Education, Handbook of Student Information System, 
Salt Lake City, Utah, February, 1970. 

This handbook accompanies the SIS manual and permits a person without special 
training to utilize SIS data in a variety of ways. Also incuded in this manual are several 
examples of research designs which are appropriate for both individual and group 
application of SIS derived data. 

United States Office of Health, Education, and Welfare, Preparing Evaluation Reports - A 
Guide for Authors, United States Government, Printing Office, Washington, 1970. 

This guide is to aid evaluators in preparing reports through decisions of what to include 
and how to report it. It follow* a very logically developed pattern consisting of the context 
of the program, the program itself, evidences of changes, and recommendations. 
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INDICATORS OF QUALITY 
adapted from a brochure printed by 

The Institute of Administrative Research 
Teachers College - Columbia University 

The Indicators of Quality is a new instrunnent for measuring school quality by 
observing the behavior that goes on in the classroom. It is based on characteristics of 
school behavior which are thought to be basic to quality. Four basic ones have been 
identified: individualization, interpersonal regard, creativity, and grc ip activity. These were 
determined by educational experts who wsre asked to decide upon .that bases they would 
judge school quality. They may be observed in the teaching-learning procedure. 

Certain key concepts were found to define ihe content of the four criteria. They are: 

Individualization 

1. Knowledge of pupils 

2. Physical facilities 

3. Different ta<<s 

4. Participation 

5. Communication 

6. Modification of questioning 

7. Complementary teacher-pupil roles 

8. Time for growth 

9. Individual evaluation 

Interpersonal Regard 

10. Demeanor 

1 1 . Patience 

12. Pupil involvement 

13. Physical movement 

14. Respect 

15. Error behavior 

16. Pupil problems 

17. Atmosphere of agreement 

18. Teacher-pupil identification 

19. Evaluation as encouragement 

Creativity 

20. Time for thinking 

21. Abundance of materials 

22. Skills for thinking 

23. Testing ideas 

24. Unusual ideas 

25. Question and answer technique 
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26. Self-initiated activity 

27. Opportunity for speculation 

28. Evaluation as motivation 

Group Activity 

29. Physical arrangement 

30. Teacher purpose 

31. Decision-making 

32. Intercommunication 

33. Conflict resolution 

34. Cooperation 

35. Role distribution 

36. Group goals 

37. Group personality 

38. Consensus 

39. Group evaluation 

40. Teacher's group role 

The instrument includes 51 items based on these 40 key concepts. Some of the 
concepts relate exclusively to teacher behavior, or pupil behavior; others may be discerned 
both in what teachers do and what pupils do. 

With a criterion of quality exhibited in this instrument, objective scores obtained can 
be statistically related to other measures that result from administrative policy: level of 
financing, teachers' salaries, teacher characteristics, staff deployment and class size, com- 
munity conditions, pupil background characteristics, pupil achievement, state legal 
regulations and a host of other factors that, in one way or another, are presumed to 
influence the quality of education. A study of these factors should indicate what changes 
could be made in the way our education is supported and managed. 
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INFORMATION COLLECTION INSTRUMENTS 



Name 



In anticipation of the training sessions in evaluation (November 23, from 1:00-4:00, 
November 24, from 9:00-4:00, and November 25, from 9:00-12:00) we are asking you to 
complete this questionnaire. You will again be grouped on the basis of your choices. Please 
return this sheet to Vickey Radcliffe, 411 Building, South 13th Street, Lincoln. 



Below is a list of educational concerns of the people of Nebraska, which might be 
considered as goal statements. Using a scale from 1-7, put a check in the box below the 
number which indicates how concerned your work should be with what the statement 
suggests. A check under #1 would mean that you are directly concerned with the 
statement, while a check under #7 would show no concern at all. 



1 


2 


3 


4 


5 


6 


7 

















1. To aid in the development of materials and programs which will help 
students prepare themselves for problems they might face in our contemporary society (Ex. 
drugs, sex education, decision making). 



1 


2 


3 


4 


5 


6 


7 

















2. To initiate surveys to determine the number and description of physically 
or mentally handicapped persons in Nebraska, so that program development and resource 
allocation can be adjusted to their needs. 



1 


2 


3 


4 


sje 


7 










1 





3. To aid loca! schools in developing programs to help each student select 



and prepare himself for the world of work. 



1 


2 


3 


4 


5 


6 


7 

















4. To help schools identify their students as individuals, bO that guidance. 



academic, and vocational programs are more relevant to their needs. 



1 


2 


3 


4 


5 


6 


7 

















5. To undertake studies to determine what "teaching competency" consists 
of and develop a system by which teachers, students and administrators can cooperate in 
the measuring of "teacher competency". 



1 


2 


3 


4 


5 


6 



















6. Survey leisure activities available in Nebraska and develop suggested 



physical education programs which would prepare students for those activities. 



1 


2 


3 


4 


5 


6 


7 

















7. To examine planning services that the State Department of Education 



makes available to those people who are considering reorganization of their districts. 



1 


2 


3 


4 


5 


6 


7 

















8. To help schools provide programs and services that will strengthen their 



local communities economically and socially. 
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Name 



FINAL EVALUATION OF TRAINING SESSIONS 
IN EVALUATION SKILLS 

Part I: 

1. Put a (+) in front of the topic which you believe will be most useful to you in 
your work. 

2. Put a (0) in front of the topic which you believe will be least useful in your work. 

3. Put a (✓) in front of the topic which you feel needed more or better coverage. 

4. Put a (X) in front of the topic which you feel needed less coverage. 

Using a Systematic Planning Model 

Evaluation and Educational Decisions 

Information Systems and Evaluation 

Indicators of Quality 

1. Put a (+) in front of the methods which you feel were most effectively used. 

2. Put a (✓) in front of the methods which were adequate. 

3. Put a (0) in front of the methods which were least effectively used. 

Media 

Notebooks 

Group discussions 

Department speakers 

Outside speakers 

Part II: 

1. What problems do you anticipate in using what you have learned? 

2. What contribution were you able to make in these sessions? 



3. What was personally most meaningful to you about these sessions? Why? 



4. If anything happened which aroused strong feelings in you (pleasant or 
unpleasant), briefly describe it. 



ERIC 
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Name 



QUESTIONNAIRE FOR EVALUATION SKILLS 



Part I: 

In the blanks preceding each statement of function, code the kind of decision and 
evaluation which that function exhibits in educational development. 

Decisions Evaluation 

Code Code 

A. Planning i. Product 

B. Structuring 2. Process 

C. Implementing 3. Input 

D. Recycling 4. Context 



Decisions Evaluation Functions 

Seeking information from parents, students, visitors and 
teachers about the impact of a Title III project and comparing 

their responses with the project's objectives. 



Seeking information from patrons and educators about the 
problems in education v/hich they believe face the people of 
Nebraska. 



Monitoring the reading progress of students involved in a 
project where the main objective is to improve their reading 
level by 2 grades. 

Comparing the proportion of students retained in school 
between the 11th and 12th grades before and after the 
employment of a guidance counselor. 

Having a representative group study literature on 
differentiated staffing, poll community attitude towards it, 
compare facilities available with those necessary to initiate it, 
and survey staff attitudes and availability. 

Identifying the factors behind student unrest. 

Measuring the learning gain from a television series. 

Conducting a site visit to an ESEA Title III project. 

Prior to initiating a humanities course, surveying available 
literature and materials, studying and seeing course objectives 
by the English Department and dr iding on interested, 
capable teachers for conducting the class. 

Interviewing parents of children involved in classes for the 
handicapped to get their opinions about program success. 
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Part II: 



Below is a list of situations in a classroom, identify the concept which the content suggests. 
Put the abbreviation of one of the four Indicators of Quality in front of each situation: 

I = Individualization 
IR = Interpersonal Regard 
C = Creativity 
GA = Group Activity 

1, Pupils have the stimulation of materials and other resources in great richness 

and variety. 

2. Pupils are cooperating in putting on a play. 

3. Pupils are seated in a circle. 

4. A pupil giving a lengthy, detailed answer is listened to and accepted by the 

class and teacher. 

5. Pupils are given time to choose books to read, while the teacher helps 

individual students pick books they are interested in and capable of reading. 

Part III: 

Put a T in front of the statements you believe to be true and an F in front of those which 
you believe are false. Use the back to explain any answers to statements which you feel are 
not clearly stated. 

; 1. To operationalize evaluation, we need objectives. 

2. There is little need to make provisions for evaluation procedures when 

planning learning activities. 

3. Some kind of evaluation is needed prior to the decision to write a proposii!. 

4. An evaluation department which is to provide information for planning 

decisions must include a component which has as its mission the development 
of an information system. 

5. The Indicators of Quality can be effectively used to evaluate the performance 

of teachers. 
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